Add support for worker pools that exclude specific queues #1148
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Nicholas Jakobsen <[email protected]>
Co-authored-by: Nicholas Jakobsen <[email protected]>
The addition of `exclude_specified_queues` allowed all specified queues to be treated as exclusions, implying that all jobs in those queues not be handled by the workers. However, this only allows application of this behaviour globally. What we want is the ability to create two or more pools, where some pools handle jobs from the given queues and other pools handle jobs that are not from the given queues. To implement this, we check if the `exclude_specified_queues` is not set and treat this as a request for automatic detection of exclusions. By adding a "!" to the beginning of a queue list, only the workers that handle that list of queues will behaves as if the `exclude_specified_queues` flag had been set.
|
Any chance this one could be reviewed and merged? |
|
@albus522 could you please review this PR? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Delayed Job already supports worker pools that draw from specific queues. This is useful to ensure those tasks are processed in a timely manner and do not starve. Coupling that with a pool that processes all jobs is a common practice. However, it is possible to create so many jobs for a specific that other jobs are delayed to an unacceptable degree. In this case we want to have a worker pool that only processes jobs that are not from that queue (or queues). There is currently no mechanism for this, hence this PR.
The changes allow worker pools to be named with a
!prefix which would indicate that we want this pool to process any jobs except those in the given queues. It is intended to be coupled with a change at the adapter level which would read theexclude_specified_queuesflag and return the appropriate jobs based on whether the list of queues is intended as an inclusion or exclusion filter.See this PR for changes made to DJ active_record adapter to support this new feature collectiveidea/delayed_job_active_record#194.